Skip to content

test(env): hold both DEPLOYMENT_SUITE values in one test, not two - #37

Merged
thedavidmeister merged 3 commits into
mainfrom
2026-08-14-issue-36-env-race
Aug 14, 2026
Merged

test(env): hold both DEPLOYMENT_SUITE values in one test, not two#37
thedavidmeister merged 3 commits into
mainfrom
2026-08-14-issue-36-env-race

Conversation

@thedavidmeister

@thedavidmeister thedavidmeister commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Closes #36.

vm.setEnv writes the forge PROCESS' environment. It is scoped to neither a
test nor a contract, no cheatcode unsets or restores it, and forge runs tests
concurrently — so two tests holding two values for one variable is two tests
racing over one variable. RainDeployBroadcastTest was exactly that, and was
seen losing the race: the unset case resolved the address-registry the other
test had written.

[FAIL: Error != expected error:
 UnknownDeploymentSuite("address-registry", "address-registry-0-0-1, second-address, address-registry-candidate")
 != UnknownDeploymentSuite("", "address-registry-0-0-1, second-address, address-registry-candidate")]
 testRunUnsetSuiteReverts()

The fix

The two values are sequenced inside ONE test, which is the only ordering forge
guarantees, and every vm.setEnv in the repo is now in that one test — the
DEPLOYMENT_SUITE write after the only read that needs it absent, so there is no
write left for another test to observe. Nothing else in the repo reads
DEPLOYMENT_SUITE or DEPLOYMENT_KEY: the only reader is run(), and the only
caller of run() is this test.

There is no smaller fix. Foundry offers no scoped or restorable env: setEnv is
std::env::set_var on the process, and unsetEnv does not exist. Making the
value injectable instead would remove the only assertion that the value comes
from a variable of THAT name (killed as M2 below), and serialising the suite with
--threads 1 would trade the whole suite's parallelism to leave the sharing in
place.

The two variables are not the same problem

DEPLOYMENT_SUITE has to be genuinely ABSENT, and that is the assertion doing
the work. vm.setEnv("DEPLOYMENT_SUITE", "") set a variable that was PRESENT and
empty, so vm.envOr's default was never reached and a default that silently
became a real suite key passed — the exact hazard that test's own comment names,
"a deploy that picks something when told nothing". Nothing in CI sets that
variable (checked: neither rainix-sol-test nor this repo's caller workflow), so
assertFalse(vm.envExists("DEPLOYMENT_SUITE")) holds and states the precondition
by name.

DEPLOYMENT_KEY is the opposite: absence is not this test's to give.
rainix-sol-test sets DEPLOYMENT_KEY: ${{ secrets.PRIVATE_KEY }} on the job
that runs this suite, so it is present everywhere the test really runs. So the
test SETS it, to something vm.envUint cannot parse. That is a stronger oracle
than absence, not a weaker one: the ordering claim is "the suite was resolved
before the key was read", and a key that PARSES makes both orderings produce the
same revert — so under CI's own key, asserting absence would have discriminated
nothing about ordering even if it had passed. Unreadable-by-construction makes
M3 die in every environment.

Mutation matrix

forge test --match-path test/src/abstract/RainDeployBroadcast.t.sol --match-test testRunSelectsTheSuiteFromTheEnvBeforeTheKeyAndNeverDefaults,
run under each ambient DEPLOYMENT_KEY state the suite meets — unset (bare
shell), set to a valid key (CI with the secret populated), set to empty (CI with
the secret absent, which still exports the variable):

# Mutation in src/abstract/RainDeployBroadcast.sol key unset key valid key empty
M0 none (baseline) PASS PASS PASS
M1 envOr default """address-registry-0-0-1" FAIL FAIL FAIL
M2 "DEPLOYMENT_SUITE""DEPLOYMENT_SUIT" FAIL FAIL FAIL
M3 envUint("DEPLOYMENT_KEY") moved above suiteByName(...) FAIL FAIL FAIL

M1 and M3 fail with Error != expected error: vm.envUint: failed parsing $DEPLOYMENT_KEY as type uint256: parser error != UnknownDeploymentSuite("", ...);
M2 with UnknownDeploymentSuite("", ...) != UnknownDeploymentSuite("address-registry", ...).

Two more results from the same matrix:

  • M1 run against the tests as they were BEFORE this PR: testRunUnknownSuiteRevertsBeforeReadingTheKey
    and testRunUnsetSuiteReverts both PASS. A deploy script that picks a real
    suite when told nothing survived the old suite; it is killed here.
  • The first commit on this branch, run under a set DEPLOYMENT_KEY, reproduces
    the CI red exactly — [FAIL: assertion failed] ... (gas: 3980), the same gas
    figure as the failing job
    — and passes in a bare shell. That is the whole of the CI failure, confirmed
    rather than assumed: the -vvv trace shows both VM::envExists calls followed
    by one VM::assertFalse(true), i.e. the DEPLOYMENT_KEY assertion.

forge fmt --check passes. forge test --no-match-contract Chain on this branch,
run with DEPLOYMENT_KEY set as CI sets it, is 92 passed / 38 failed where all 38
are vm.createSelectFork: environment variable *_RPC_URL not found in
LibRainDeployTest — what #32 is about and #34 fixes. This branch changes one
file and it is not that one.

QA

  • Discriminating tests: testRunSelectsTheSuiteFromTheEnvBeforeTheKeyAndNeverDefaults
    — passes unmutated (M0) and fails under each of M1, M2 and M3, in all three
    ambient DEPLOYMENT_KEY states (unset, valid, empty): 12 runs, one per cell of
    the matrix above. Each is its own forge test --match-path test/src/abstract/RainDeployBroadcast.t.sol --match-test <name> with the [PASS] / [FAIL: ...] line captured, so every
    cell is a run that actually selected and executed the test rather than a filter
    matching nothing. The two tests this PR does not touch —
    testDeployNetworksDefaultsToSupportedNetworks, testSelectedSuiteCarriesTheRecordedPins
    — are unchanged.
  • Mutations applied (all in src/abstract/RainDeployBroadcast.sol, the code
    under test; killer is the test above in every case, in every environment):
    • M1 — vm.envOr("DEPLOYMENT_SUITE", string(""))string("address-registry-0-0-1")
      → KILLED: vm.envUint: failed parsing $DEPLOYMENT_KEY as type uint256 != UnknownDeploymentSuite("", ...).
      This mutant SURVIVES the pre-PR teststestRunUnknownSuiteRevertsBeforeReadingTheKey
      and testRunUnsetSuiteReverts both PASS under it. That gap is what the new
      test closes.
    • M2 — "DEPLOYMENT_SUITE""DEPLOYMENT_SUIT" → KILLED:
      UnknownDeploymentSuite("", ...) != UnknownDeploymentSuite("address-registry", ...).
    • M3 — uint256 deployerPrivateKey = vm.envUint("DEPLOYMENT_KEY"); moved above
      the suiteByName(...) line → KILLED: vm.envUint: failed parsing $DEPLOYMENT_KEY as type uint256 != UnknownDeploymentSuite("", ...).
      Killed under a VALID ambient key too, which is the case that matters: this
      mutant is invisible to any test that lets a parseable key reach envUint,
      because then both orderings revert identically.
  • Oracle: the workflow's contract with the script, not the implementation.
    Manual sol artifacts hands the script a suite key as DEPLOYMENT_SUITE and key
    custody as DEPLOYMENT_KEY, so the expected behaviour is fixed independently of
    the code: the key the caller passed is the key reported back, passing no suite
    selects no suite, and neither answer requires a usable private key. The
    valid-suite list in the expected payload is the fixture's own declaration
    (ExampleDeploySuites), asserted separately by
    RainDeploySuitesBaseTest.testSuiteNamesIsTheRegistry.
  • Category check: Flaky suite: two tests write the process-global DEPLOYMENT_SUITE, so one reads the other's value #36 asks for the shared vm.setEnv state removed without
    either assertion getting weaker, and names the three behaviours the result must
    still discriminate — the envOr default becoming a suite key, the variable name
    changing, and the key being read before the suite. Covered by M1, M2 and M3
    respectively, plus the sharing itself: every vm.setEnv in the repo is in this
    one test, with the DEPLOYMENT_SUITE write after the only read that needs it
    absent.

`vm.setEnv` writes the forge process' environment: scoped to neither a
test nor a contract, restored by nothing, and read by tests forge runs
concurrently. Two tests each writing that variable is two tests racing
over it, and the unset case was seen resolving the `address-registry`
the other test had written.

Sequencing both values inside a single test is what removes the sharing
— it is the only ordering forge guarantees, and there is now exactly one
write to that variable in the repo.

Unset is now genuinely unset rather than `setEnv("", ...)`, which set a
variable that was present and empty and so never reached `vm.envOr`'s
default. Both absences are asserted, because both are inputs: an unset
`DEPLOYMENT_SUITE` is what makes the empty key report a missing default
rather than an empty value, and an unset `DEPLOYMENT_KEY` is what makes
the revert say the suite was resolved first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@thedavidmeister thedavidmeister self-assigned this Aug 14, 2026
@coderabbitai

coderabbitai Bot commented Aug 14, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@thedavidmeister, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 93 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: d55082f4-5563-4b04-ac59-bde96ef76341

📥 Commits

Reviewing files that changed from the base of the PR and between 2317e49 and e02a421.

📒 Files selected for processing (1)
  • test/src/abstract/RainDeployBroadcast.t.sol

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: e372f17c-9f88-4a4d-9e24-2baa830e78e9

📥 Commits

Reviewing files that changed from the base of the PR and between a7e7cb2 and 2317e49.

📒 Files selected for processing (1)
  • test/src/abstract/RainDeployBroadcast.t.sol

Walkthrough

The change replaces two deployment suite validation tests with one sequenced test. The test checks unset and invalid DEPLOYMENT_SUITE values, lists valid suites in both errors, and verifies suite resolution before DEPLOYMENT_KEY access.

Changes

Deployment suite validation

Layer / File(s) Summary
Sequenced suite validation
test/src/abstract/RainDeployBroadcast.t.sol
The two suite-selection tests were consolidated. The test verifies initially absent environment variables, unset and invalid suite errors, the complete valid-suite list, and validation before DEPLOYMENT_KEY access.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 2317e

This PR sequences shared deployment-suite environment values within one test and strengthens coverage of unset, variable-name, and evaluation-order behavior; no actionable merge-blocking risk remains beyond normal checks and review.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The changes address issue #36 by sequencing both scenarios and preserving checks for unset suites, variable names, valid suites, and key-read order.
Out of Scope Changes check ✅ Passed The changes are limited to consolidating and strengthening the affected suite-selection tests.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: consolidating both DEPLOYMENT_SUITE scenarios into one test to avoid concurrent environment-variable writes.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch 2026-08-14-issue-36-env-race

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

thedavidmeister and others added 2 commits August 14, 2026 17:17
`rainix-sol-test` exports `DEPLOYMENT_KEY: ${{ secrets.PRIVATE_KEY }}`
onto the job that runs this suite, so the variable is PRESENT everywhere
this test actually runs and `assertFalse(vm.envExists("DEPLOYMENT_KEY"))`
was red in CI and green in a bare shell.

Absence was the wrong input to reach for anyway. The ordering claim is
"the suite was resolved before the key was read", and a key that PARSES
makes both orderings produce the same revert — so under CI's own key the
assertion would have discriminated nothing even had it passed. The test
now SETS the key to something `vm.envUint` cannot parse, which is an
input it controls and which makes the ordering observable in every
environment.

`DEPLOYMENT_SUITE` still has to be genuinely absent — that is what keeps
`vm.envOr`'s default under test rather than restating an empty string —
and nothing in CI sets it, so that assertion stays.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@thedavidmeister
thedavidmeister merged commit 8c2662a into main Aug 14, 2026
4 checks passed
thedavidmeister added a commit that referenced this pull request Aug 15, 2026
`test/src/abstract/RainDeployBroadcast.t.sol` is the only conflict, and both
sides changed the same two assertions for unrelated reasons.

Main (#37) folded `testRunUnknownSuiteRevertsBeforeReadingTheKey` and
`testRunUnsetSuiteReverts` into one test, because `vm.setEnv` writes the forge
PROCESS' environment and two concurrent tests holding two values for
`DEPLOYMENT_SUITE` were racing over one variable. The unset half has to run
before anything sets the variable, and asserts `vm.envExists` is false as its
precondition.

This branch added a second candidate, so the suite keys a mistyped
`DEPLOYMENT_SUITE` reports grew a `second-address-candidate`.

Both survive: main's single sequenced test, with the four-key list in BOTH
expected reverts. The branch's `testRunUnsetSuiteReverts` is dropped rather
than carried — it is the very `vm.setEnv("DEPLOYMENT_SUITE", "")` main removed,
and keeping it would both restore the race and falsify the `envExists`
precondition of the test that replaced it.

Auto-merge had already taken main's structure while updating only the SECOND
key list, leaving the first assertion naming three suites; that is corrected
here, not inherited.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky suite: two tests write the process-global DEPLOYMENT_SUITE, so one reads the other's value

1 participant